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ABSTRACT 



A data structure that uses one or more keys which are 
members of a multidimensional composite key to partition 
sparse data (FIG. 2A, FIG. 2B) into a hypercube like 
structure (FIG. 1). Only the data structure produced by the 
method in this invention is actually better than a hypercube 
since all involved keys can have true symmetric access as 
opposed to just the keys that have the highest sort order 
significance. Equivalent query efficiency could only be 
otherwise obtained through storing multiple copies of the 
sparse data sorted on different permutations of the member 
keys of the multidimensional composite key (FIG. 2C). The 
data structure has two major components: the partitioned 
data with or without the boundary keys and the non-dense 
index necessarily containing the boundary keys (FIG. 3). 
This data structure and partitioning method can be used to 
partition multidimensional data across physical hardware 
storage devices and parallel computer architectures (FIG. 4) 
to produce variable grained data partitions that can be 
queried, loaded, or updated in parallel and simultaneous 
multidimensional index selectivity. And finally, as noted 
above, this same partitioning method can be used in a nested 
fashion or in combination with other data structures to 
provide good performance for the batch maintenance of 
volatile data and special performance advantages for true 
time-series data. 

6 Claims, 5 Drawing Sheets 
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INTERVAL-PARTITIONING METHOD FOR The problem literally can cause a query to run for days. 

MULTIDIMENSIONAL DATA Ironically, via indexes, the query can end up requiring much 

more time than a single full tabic scan would have required 
BACKGROUND in the first place. 

1 Field of Invention 5 ^ e }SSC °^ mc * exes tDat ^ m *°*ged as heaps in combi- 

. . nation with tables that are managed as heaps makes it 

This invention relates to OLAP (OnLine Analytical difficult to mdexed access ^ parallel query 

Processing) and Data Warehouse applications speafically processing . since records that have equal key field values 

to data structures and algorithms providing better pcrfor- can randomly ^ storcd in the heap> it Ls difficult to partition 

mance and efficiency for these apphcations by providing a JQ the table and index data into component pairs that can be run 

very effective storage and retrieval mechanism for multidi- independently 

mensional data. M if ^ Qther problems werc not enougnj me solution 

2. Description of Prior Art also wastes too much space. Data must be stored in indexes 

Data Warehouse and OLAP applications have highlighted as well as tables. Albeit, some indexes are more frugal in 

the need for a fast way to store and retrieve multidimen- 15 their use of space than others. Any duplication of data, 

sional data. To date other attempted solutions have not met however, is magnified by the fact that tables of this type can 

this need. range into several terabytes in size. 

To illustrate the prior art and how it differs from this While this solution is excellent for OLTP applications, it 

invention consider sales data dimensioned by zip code, day, is clearly less than optimal for OLAP applications, 

and product. Sales data is stored for 50,000 zip codes, 1000 20 2) Take advantage of modem hardware technology and 

days, and 10,000 products. The sparsity factor is 1% on store the data in a non-indexed table and allow parallel 

average so that of the 500,000,000,000 records that would processors and high capacity I/O subsystems to do the work, 

result from all possible combinations of zip code, product, Obviously, this is too inefficient. No matter what degree 

and day only about 5,000,000,000 records actually have of parallelism is used, queries in the example above should 

associated data and need to be stored in the fact table. In 25 not nave to access 65 gigabytes of data to retrieve a 10 

addition to the keys, the record format contains a field for kilobyte result set. 

sales so that the record format is represented by FIG. 2 A and If Ws or 100 » s of m mninng these type queries on 

FIG. 2B below. one or more tables, database server resources can be quickly 

Even with a very efficient table and index design, the depleted, 

space required to store only the records with actual data is 30 This solution causes queries to run too long and uses too 

at least 5,000,000,000x(2 bytes+2 bytes+2 bytes+4 bytes+4 maoy resources. The data should be organized in such a way 

bytes) or approximately 65.19 gigabytes. mat data accessed together is stored together. Then queries 

Couple this with the fact that users of the data may want can perform I/O only for the table pages that are needed to 

to query the data by any combination of zip code, product, satisfy the query. 

and/or day. They may want the total sales dollars for a given 35 3) store the data in a cube-like data structure as depicted 

product and day. Or they may want to know the average m FIG. 1 below. Many niche OLAP products use a similar 

sales volume per month in a certain zip code for the past 12 structure. 

months. Any combination of key fields along with any If the data were not sparse, this would work pretty well, 

possible values of those key fields may be chosen. Hie data It would provide effidcnt ^ fast access to lhe data ^ the 

truly must be multidimensional with efficient and symmetric query acccss would t^ fairly symmetrical. But, with the 

query access paths. sparse sales data in the example above, approximately 6.5 

There is not a good way to store and retrieve this data with terabytes of data would be required to store a cube large 

today's database technology. Consider the alternatives: enough to house the data. This would be 100 times larger 

1) Store the data in a heap table (as is commonly done in 45 and 100 times slower in terms of I/O to access than the 

relational databases) and use some combination of one or original data. In addition, a more subtle problem emerges. If 

more indexes. the order in which the data is stored in the cube is by zip 

The type of index does not matter. They could be tradi- code then product and finally time, access may not be so 

tional B-trees, more sophisticated star-join indexes, bit symmetric. Access for the outer-most key fields is fine but, 

maps, and/or hashing routines. No matter how efficient the 50 restricting the data by choosing a small subset of days from 

index is, all table pages that contain data satisfying the query the possible choice of 1000 days may not yield improvement 

must be accessed at least once in order to produce a query over a full table scan in query performance. If the data is 

result set. If the data for a zip code, for example, is spread stored in a database with 16 kilobyte table pages, then to 

thinly over many table pages as opposed to clustered or access all the sales data for one day, all data in the entire fact 

grouped together in fewer pages, query performance is 55 table would have to be scanned because an occurrence of 

degraded by the additional I/O that is necessary to retrieve that particular day would exist on nearly every table page, 

the extra table pages. This is in contrast to the fact that the data for one day should 

In addition, as many developers and DBAs know from constitute about l/1000th of the size of the complete fact 

experience, the problem can be much more severe than this. table. 

A combination of poor optimizer access plans and the use of 60 Th^ solution is not realistic for sparsely populated OLAP 

dense indexes with low data cardinality often causes more f act tables. Attempts have been made to pack sparse fact 

harm than good. The optimizer plan can access a given table tables into smaller cubes but, this further limits the sym- 

page many times during the life of the query in lieu of metric or multidimensional capabilities of the cube, 

extracting all rows satisfying the query during a single pass 4) Use clustering and/or partitioning to "divide and con- 

of the table page. This might be true in the business example 65 quer" the data. 

above if the data in the table were sorted by product and all This method works pretty well as long as the data is 

the data for a group of days for all products were queried. clustered or partitioned in the same way that it is accessed. 
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But, query performance is not symmetric with this solution. 
If the data is accessed in a way that is different, performance 
can be as bad or even worse than storing the data randomly 
or in a heap like structure. This is true since clustering or 
partitioning the data by one set of keys, distributes or 5 
disperses data with the same key values that are not part of 
the cluster or partition or are not in the prefix of the 
clustering or partitioning key. 

Multiple partitions and clusters can be constructed for the 
same data. This improves query performance since it pro- 10 
vides the clustering or partitioning advantage for a wider 
range of queries. However, each new cluster or partition 
requires an additional copy of the complete table. Of course 
this is very expensive in terms of disk space. And in order 
to make this approach completely general so that queries 15 
involving any combinations of key fields from the composite 
key of the table can take advantage of the technique, an 
exponential number of clusters or partitions with respect to 
the number of fields in the key must be built. To be 
completely effective, each of these clusters must replicate 20 
the data. 

For generalized, symmetric OLAP access, this method is 
less than ideal. 

5) Use a special approach just for OLAP or GIS/spatial ^ 
data that combines the best part of the previously mentioned 
methods. 

The most successful method of this type to date has been 
the grid file. 

The grid file actually works very well in terms of query 30 
efficiency since it clusters data based on all key fields in the 
composite key of an OLAP table. Therefore, no matter 
which combination of key fields and values within those key 
fields are used to qualify a query, the grid can be used to 
narrow the number of table pages that must be accessed in 35 
order to retrieve the query set. In addition, grid file indexes 
or scales are usually designed small enough so that they can 
be stored entirely in RAM memory to further aid query 
speed. 

Never the less, grid files are not without problems. The 40 
way in which data is divided leaves large holes in the storage 
space thus wasting space and slowing queries down by 
requiring more I/O. 

To compound this, grid files also usually store the actual 
table pages in a heap like fashion. This makes the combi- 45 
nation of grid files with parallel processing more difficult to 
achieve. 

This method also leaves much to be desired but, there is 
a better way. 50 

OBJECTS AND ADVANTAGES 

This invention has been specifically designed for OLAP 
multidimensional tables. As a result, it solves the problems 
of the five major approaches mentioned above and provides 55 
one very effective method for storing OLAP data. 

It does not have the problems associated with the first 
approach above. It stores the table and indexes all in one data 
structure. The space requirements for this data structure are 
no larger than the requirements to store only the table in 60 
conventional approaches. No space is needed for indexes. In 
fact, table pages are packed 100 percent full so that this 
complete data structure (data and indexes) requires much 
less space than most heap tables by themselves. It also does 
not have the problem of dual I/O (index and table). And, the 65 
data is organized according to the algorithm or method 
contained in this invention. This organization is similar to 
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that of clustering or partitioning except that is truly sym- 
metric and multidimensional. Amazingly, this data structure 
still allows efficient incremental updates. This is especially 
true for time-series data where incremental updates for this 
data structure are very efficient. 

This data structure is easy to be divided into parallel 
components, as in the second approach above, while avoid- 
ing the problems of depending only of parallel technology. 
It is not necessary to query each combination in the Carte- 
sian Product space of all values selected from each dimen- 
sion in order to perform a star-join. These individual dimen- 
sion value lists can be divided up to match the organization 
of this data structure. Each of these subcomponents of the 
query can be queried, grouped, and aggregated indepen- 
dently. This data structure allows the query to divided into 
a granularity that matches the number of logical disks that 
the data is stored across. RAID technology such as RAID -5 
actually improves the efficiency of the data structure by 
making it more symmetric. With RAID technology, queries 
using any combination of key field values, execute as if all 
data needed by the query is stored contiguously. The data 
structure also issues all I/O at the beginning of the query and 
in disk page order so that thrashing of the disk head is 
prevented. But most important of all, the data structure 
allows the query to select only the table pages containing 
data needed by the query at the same time that it takes full 
advantage of parallel technology at a maximum level of 
granularity. 

Data is stored in a similar manner as in FIG. 1 below. This 
invention has all the advantages of a hypercube even when 
the data has very high levels of sparsity. For high levels of 
sparsity the symmetric and multidimensional qualities of 
this data structure are not diminished yet, the data is com- 
pacted to occupy a minimum amount of space. Moreover, 
this invention is actually better than a hypercube since it 
preserves multidimensionality no matter what order the key 
fields data are stored. Still further, the degree of multidi- 
mensionality and selectivity of key fields can actually be' 
adjusted to suit individual application needs. 

This data structure provides the ultimate data locality 
sought after with the fourth approach of clustering and 
partitioning. This data structure provides performance vir- 
tually as good as if the data were sorted and partitioned by 
every possible permutation of key fields in the table. But, 
instead of requiring the data to be replicated a large number 
of times, this invention does not require any space for 
indexes. Using this data structure in the business example 
above is virtually the equivalent of replicating and sorting 
the data by all the key permutations in FIG. 2C below. Of 
course this data structure requires no replication. 

Finally, this invention is more efficient than the grid file 
data structure in the fifth approach. No space is wasted when 
the data is divided by grid scales. All table pages are always 
100 percent full. In addition, no additional space is need for 
indexes. And, as mentioned before, it is well suited toward 
parallel queries. 

This invention truly has the ability to revolutionize the 
data warehouse and OLAP industry. It is simple to imple- 
ment and maintain yet provides very powerful features 
designed just for this type of data. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a type of data structure contains place holders for 
records for every possible combination of keys in a multi- 
dimensional file. 

FIG. 2A and FIG. 2B represent data that might be stored 
in a multidimensional data structure. 
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FIG. 2C represents a minimum set of sorted copies or With this information, the number of magnetic disk 

permutations of the data that would be required to efficiently resources utilized can be calculated. This fact table contains 

query on all possible multidimensional key combinations if 1 6x250,000,000 bytes or about 3.72 gigabytes of data. If this 

the sample data in FIG. 2A and FIG. 2B were stored in data is to be stored on a disk drive unit with track sized 

B-tree type data structures. 5 partition units of 50 kilobytes, 78,125 disk tracks would be 

FIG. 3 is the internal data structure of a typical embodi- required to store the data. In this case disk tracks are chosen 

mcnt of the Hypergrid. to be the unit partition. Disk sectors or pages of other various 

„ r AA CTr ATt rnn An xnn at\ „^ vnn Ar. sizes could also have been chosen as the unit partitions in 

FIG. 4A, FIG. 4B FIG.-4C, FIG 4D aod FIG^ 4E 5Q g ^ 

represent some typical embodiments of the Hypergnd on (input/output) roUtini storage technology, 

vanes computer hardware configurations. 10 . . . . v f. . f ,j - « . , • , . * 

F & the size of a unit partition should fall inclusively between a 

DESCRIPTION — MAIN EMBODIMENT disk sector and disk track so that data in a unit partition can 
The method contained in this invention partitions multi- *>c accessed with exactly one physical I/O operation without 
dimensional data to provide efficient query access for Data moving the read/write head. Since the index associated with 
Warehouse and OLAP (Online Analytical Processing) 15 this data structure is non-dense and an order of magnitude 
applications. OLAP data can be conceptually represented by smaller than the data, it should be possible in most cases to 
a multidimensional or hypercube structure as depicted in retain the index portion of a Hypergrid in RAM (Random 
FIG. 1. OLAP data can only be stored in such a hypercube Access Memory) for maximum efficiency and speed. In 
structure if none or only a very small portion of the data is addition, any record in the Hypergrid should be accessible 
missing or in other words if the sparsity factor is low. If data 2jo ^ onl y one 1/0 U multi P le records in the query criteria 
with a high sparsity factor is stored in a hypercube, a large are on one unit partition, that unit partition should be visited 
amount of space is wasted. In short, the Hypergrid allows no more than one time during the entire query operation, 
sparse data to be stored in a multidimensional type data The Hypergrid partitions the data using all fields in the 
structure such as depicted in FIG. 1. This is done, as the title composite key that have been selected as member fields, 
suggests, by sorting and partitioning the data into intervals 25 This can be any subset of the composite fields. Only fields 
of values. Instead of sorting and organizing the data com- that are member fields of the composite key can be used by 
pletely by one dimension, it is sorted and partitioned accord- the Hypergrid to selectively eliminate unit partitions that do 
ing to a controlled number of intervals for each dimension. not need to be searched. This is a result of the way the data 
For example, if the data to be organized contains 1000 is structured or partitioned in the Hypergrid. This partition- 
distinct product values and 5000 distinct time values, it 30 ing can be done in several ways. The number of partitions 
would not be sorted by product into 1000 different groups or can be divided evenly among the member fields. In the case 
alternatively by time into 5000 different groups. Instead, it of this example, there would be approximately 42 partitions 
might be sorted and partitioned into 50 equal sized intervals per field. Note that more than 50 partitions for the state key 
of product values and, within those intervals, 250 intervals should not be used since there are only 50 states and more 
of time values. In this way, each dimension assists in the 35 than 50 partitions would not provide additional benefits, 
organization of the data and can thereby be used in the query Alternatively, the fact table could be analyzed in full or 
process to eliminate partitions or segments of the data which estimated to determine the relative number of distinct data 
do not need to be searched for a particular query. This is true values or relative cardinality for each member field in the 
no matter which combination of dimensions and selection composite key compared to the other member key fields in 
criteria are used to query the data. The structure and com- 40 the composite key. In this case there are likely to be about 
ponents of a Hypergrid are defined with the aid of the 50 times more products than days and about 20 times more 
example below. days than states. Therefore, the product field would be 
A few assumptions are needed to begin the example. A assigned about 1000 partitions, the time field would be 
fact table is a set of data based on a composite or compound assigned 20 partitions, and the state field would only be 
key. The composite key is usually, but not necessarily, made 45 assigned 4 partitions. These partitions could also be assigned 
up of integers. An example is depicted in FIG. 2 A. Each using fixed, relative, minimum or maximum weights. For 
component key or field in the composite key of the fact table example, the product field could be constrained to have at 
corresponds to a single field primary key in a dimension least twice the number of partitions as the time field. As 
table. The relationship between each component key in a noted above, each field or component key should always be 
fact table and a primary key in a dimension is commonly 50 constrained to have no more partitions than there are distinct 
known as a foreign key relationship in databases. values for that key which occur in the data. In summary, the 
Hie example in the previous paragraph only depicts a few number of partitions for each field could be determined 
records. But, consider the same type data expanded to "V combination of analytical and explicit assign- 
include 50 states and 50,000 products. Assume data is ments to ■»* fit ** &™ application or situation, 
collected on a daily basis as opposed to a monthly basis. 55 The partitions are used to structure the data in the fact 
Further, assume that about 3 years or 1000 days worth of table. These partitions are used to divide the data across the 
data is maintained. With low sparsity, this fact table could d«k tracks, or more generally unit partitions, that are 
contain as many as 2.5 billion records. Alow sparsity factor required for storage of the data. In the case cited above, 
for a fact table means that all or most possible combinations 78,125 would be required. For simplicity, assume the first 
of composite key values are present in the fact table. Assume 60 method is chosen and that each key partitions the data 42 
the data is 90% sparse or empty (contains only 10% of the ways. 

possible composite key values) so that there are approxi- A Hypergrid is structured as a hierarchy of partitions as 

mately 250 million records. According to FIG. 2B this data depicted in FIG. 3. In FIG. 3 each non-dense index repre- 

has a three field composite key and one metric or non-key sents a dimesion. 

field for a total of four fields. Each of these fields can be 65 The order of the member fields chosen from the compos- 
represented by a 4 byte integer so that the total number of ite key determines the order of partitioning in the Hypergrid. 

bytes per record is 16. If the order of the fields in the above example is state, 



6,003,036 

7 8 

product, and time, then the data is fiist divided into 42 devices as depicted in FIG. 4E. In this case, each group of 
partitions according to state. In order to create the first 42 RAID devices is treated as a single disk device. Instead of 
data partitions, the data can be sorted according to state, disk sectors, tracks, and cylinders, the devices usually have 
product, and time since this is the field order of the com- virtual sectors, tracks, and cylinders which work in a similar 
posite key. It is possible and probable that the data will not 5 manner as standard disk devices only with faster transfer 
change state values exactly on the partition boundaries. In times. The data partitions could also be nested. The data 
other words, all the North Carolina data will not be in one could be partitioned across non-shared computer systems 
partition while all the South Carolina is in another partition. and then within these computer system partitions, data could 
Both the North Carolina and South Carolina data would be partitioned across disks or other hardware I/O devices, 
likely overlap two or more state partitions. Consequently, 10 Each disk or I/O device could be further partitioned into disk 
other key values arc needed to divide the data exactly at the tracks or pages. The data could be still further partitioned 
partition boundaries. This is also done according to the into disk sub-pages with each disk page or track. The 
ordering of member fields in the composite key. The first Hypergrid has an important advantage in this area. Since the 
partition boundary in the 42 state partition boundaries might partitions are filled to nearly 100% of capacity and non- 
be state«2, product«2, and time=322 (Note that only the key 15 dense indexes with record promotion are used, no space is 
values are used in the table. A lookup in the dimension wasted. This is true, as well, when the partitions are nested, 
would be required to obtain the state or product name.). Thus Standard transaction B-trees and Grid files which are par- 
each of these 42 partitions has values for each component tially to completely full would waste much space if they 
field of the composite key. It is also worth noting that all were nested. For each additional level of nesting used the 
keys fields are used to determine exact partition boundaries 20 s P ace utilization would continue to decline for these other 
even if only a proper subset of the key fields are selected as data structures. 

member fields. The key fields corresponding to the last When multiple disks devices or RAID units, as the case 

record in each partition are promoted to the non-dense index may be, are used to store Hypergrid data, the data does not 

associated with this level in the partition hierarchy. Pointers have to completely fill each device. This is known as thinly 

are not needed if fixed length integers or byte fields are used 25 spreading the data. Thus, each disk could be filled to 25% 

for key values and the non-dense indexes for each level of capacity. Of course, the advantage to doing this is more 

the partition hierarchy are stored contiguously so that parallel I/O devices working on the same amount of data and 

addresses can be implied or calculated. Now within each therefore resulting in faster query (and load) times, 

state partition there will be 42 product partitions. Again, The partitions for a given key level in the hierarchy do not 

these probably will not be divided exactly by product key. 30 usually contain the same number of partitions. Consider the 

Consequently, in these 42 product sub-partitions the entire example above. Each level has 42 partitions and there are a 

composite key is once again used. In this case the order of total of 78,125 unit partitions. 42 raised to the third power 

the key is nearly the same as the parent partition but the should equal the number of unit partitions or 78,125. But 42 

leading key is moved to the last position in the composite raised to a power of 3 equals 74,088. The number 42 was 

key so that the order is now product, time, and, state. This 35 obtained by evaluating the third root (since there are three 

process recurs in all levels of the partition hierarchy from the supporting component or member key fields) of 78,125 

least granular partitions down through the unit partitions or which approximately equals 42.7 and truncating everything 

in this case disk tracks. The partitions associated with the to the right of the decimal point. 43 could not be used since 

lowest level key component also need not contain, pointers 43 raised to the third power is 79,507 which is too large, 

to unit partitions if can be implied or calculated as would be 40 Thus 42 partitions per level must be the starting point and 

the case when the unit partitions are stored contiguously on additional partitions are added where needed, 

a particular I/O device. To match the number of partitions in each level with the 

For each partition level in the hierarchy, there must be a total number of unit partitions, partitions are added in 

non-dense index. It is important to note here that a B-tree or selected sub- trees of the overall partition hierarchy tree. The 

a B+tree could be used. This coupled with only a very small 45 process of adding partitions begins at the top of the partition 

amount of free space on data pages or partitions (usually less hierarchy. Provided that the number of unit partitions pro- 

than 1 % depending on record sizes relative to partition sizes) duced by the partition hierarchy does not exceed the number 

results in the fact that very little overhead storage space is of unit partitions required by the data, the number of 

needed for the Hypergrid method. The indexes are non- partitions in the top level is increased by one. Note that this 

dense because the underlying data is sorted according to the so adds a complete sub-tree. The process is repeated for the 

index and only one index entry is needed per data partition. next lower level. This process is repeated until the bottom 

This is surprising since a non-dense index can usually only level of the partition hierarchy is reached or until no level 

be applied to one set of fields and one ordering of the data can be completely increased by one without exceeding the 

and therefore can only be one dimensional. A Hypergrid number of total unit partitions. Note that all partition levels 

allows non-dense B-trees for all dimensions, indexes Com- 55 can not be increased by one because if this were the case, the 

pression could also be used. initial starting numbers could have been larger. At all levels 

In the above example a disk track is used to partition the other than the top level, all groups of sub-partitions, accord- 
data. The partitioning can occur across all types and levels ing to partitions in the parent level, are increased by one in 
of peripheral I/O storage devices. In addition, the partition- order to increase the overall level by one. This is equivalent 
ing can be nested. As suggested before, the level or type of 60 to increasing the number of partitions in the top level by one. 
device used to partition the data is referred to as the The process is continued at each level other than the bottom 
partitioning unit. The data could be partitioned across disks level until another level can not be fully increased by one 
as shown in FIG. 4A, complete non-shared parallel com- partition in all groups of sub-partitions without exceeding 
puter systems as shown in FIG. 4B, disk sectors or pages and the number of unit partitions required by the data. Once this 
sub-pages as shown in FIG. 4C, or disk tracks and/or 65 point is reached the lowest level groups of sub-partitions can 
cylinders as shown in FIG. 4D. The data can be partitioned be individually increased by one partition each until the 
across RAID (Redundant Array of Independent Disks) I/O number of partitions in the hierarchy is exactly equal to the 
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number of unit partitions required by the data. The number For the permutation of key fields chosen in step 1), the 

of partitions at each level and the number of partitions in the number of partitions for each field must be chosen. The data 

bottom level with an extra partition arc also stored in data can be analyzed first. This will determine the key cardinality 

structure for use in queries. (The storage of the number of or the average number of distinct values that exists in the 

partitions is a requirement when addresses are implied in the 5 data for each field with the context of other field values. If 

absence of pointers.) me permutation to be used is a,b,c, then analysis provides 

Records do not align perfectly to partitions in most cases. information about the average number of field values in the 

If partition units are 50 kilobytes each, and records are 28 data f or the context of each distinct value of b and c. It also 

bytes each, 1828.57 records can be placed on each partition. provides similar information about the average number of 

For performance reasons however records should not be 1Q va i ues for b and c. OptionaUy, the analysis of key cardinality 

split across pages^ especially if an I/O is required to obtain can ^ rformed M part of ^ and partitioning of 

each page. Therefore, 1828 records must be placed on each ^ ^ ^ G r as a first step, the partitioning 

* )a ??* * . . . . - . . ■ , , , method must be selected. This may involve any combination 

The problem * worsened by variable length records and f ^ foUowi metho(Js ^ fc tQ ^ 

fields. For variable length records, a null record equal to the , . 4 . 4 - 4| _ . - 

maximum record length must be left vacant on each unit 15 P artlh ° mn g- n * rao ' of the n T ber ° f ^ 

partition. In the case of partitions internal to unit partitions P™™ required by the data where n « the "umber of 

which can be accessed in one physical I/O, this is not a meraber fie,ds ,n ** permutation that has been selected. The 

problem since sub-pages do not have to be of uniform size. n*Mttg number of partitions for each key is truncated to 

This also does not present a problem for nesting and multiple whole numbers. .„.,.. 

levels of partitions since the extra space need only be left 20 2 ) re ' alive constraints: each fleld in the permutation is 

vacant one time per physical I/O unit or unit partition. This ""go? 1 a re ! aUve constraint factor. For example 4a, lb, 3c 

wasted space could be eliminated altogether but, records mi 8 ht **= assigned. The relative constraints are determined 

may require an extra I/O to be accessed. Finally, the fact that boas the "^«» P hase - 006 ° f * e constraint factors •» 

one record per partition is promoted to the indexes, com- usuaU y ^ to L The othefs 316 ass, S ned t0 66 multiples. A 

pensatcs for the vacant space since it allows space equal to 25 P™*"* of me constraints is calculated. In this case the 

one more record per page. P 10 *"* would te 12 Thc number of unit Petitions is 

Each partition in a Hypergrid should be visited exacdy ***** ^ i? e P 10 ?™ 1 ' Th » *'h e ^justed number of unit 

c ii -j *u * Tj j • .u * at/ partitions. Then the root of the adjusted unit partitions is 

once for all records that would reside m that portion of the , , , • . * f . * ,^«. 

• , * . c ~ ,i c ii calculated as in the first method. The result of this calcula- 

data during executions of queries. Consequently, for all . . , , , 4 t , . 4 4 . 

records in that portion of the data, exacUyot?e I/O in total is 30 u ° n * «"* ^t- The adjusted root is truncated to a 

required. When more recoros are accessed per partition, the ° u ^ r - ™° °™iber of partitions for each field is the 

query becomes more efficient in terms of I/O since an P 1 ^ 1 of me ■ d J usted 1001 and each rcspecUve constramt 

individual unit partition should never be accessed more than a *L^ , . . . „ , f . . 

once in the fife of a query. Partitions not involved in the 3 > fi * ed 00118 one or ™ re fi eId *. * the peculation 

query are not accessed at all. Nested partitions, to be 35 ^ assigned a fixed constramt Assumiog that there are 

discussed in more detail below, can be used to provide ^cient unit partitions (die product of the fixed I constraints 

simultaneous multiple data partition granularities for the ™ l * la /S er man me , nu R mb ! r ° f " m f artltl0ns ). ea ch 

i . , J. . , , ~ 7 held that has been assigned a fixed constraint ls assigned that 

same data at no addiUonal cost. Therefore, once a partition , . ... 27 . . j. . , , 

is moved into memory, a processor need not scan the entire °™ ber °^^T^ """"^ °. P ^ T fT^ 

partition to locate a particular recoid. This allows unit 40 by the product of me fixedconstramtsisus^ to c^culate the 

partitions to be much larger as in the case of a disk track. As ^be' of partitions for the remaining fields. All results are 

T. . . , . , , truncated to whole numbers, 

a result B-tree index sizes are much smaller and can easily . ,. . „ , , , . , . - 

be placed in RAM memory. In addition, data can be parti- *> fields can * ^ ^ ^ e ? Um !^ f 

tioned across complete computers, disk devices, and disk ^ P^? 011 * for ea f h field is tenUUyely calculated The 

t t »i * T i 45 assigned limits are then applied. The total number of unit 

tracks all at once through nesting. rT . ...... , 

partitions is divided by the constrained fields to determine 

Operation — Main Embodiment the adjusted unit partitions and the remaining field partitions 

Construction of the Hypergrid are determined from the adjusted unit partitions. The root of 

The steps detailed below correspond to the data flow the adjusted unit partitions is also truncated to a whole 

diagram that follows them. 50 number. 

Step 1) in constructing a Hypergrid for a fact table is to When any of the above methods are used, numbers of 

determine what composite key to use in the Hypergrid for partitions are calculated. These calculations are rarely lim- 

the fact table. The key will be some permutation of the ited to whole numbers. As mentioned before, results of the 

composite key in the fact table. The fields selected in this calculations are truncated to whole numbers. Therefore, to 

permutation are known as member fields. For example, if the 55 fully utilize all partitions, the following technique is 

composite key involves fields a, b, and c, then the following employed: 

are possible permutations or member fields sets: 1) If possible, and the product of the partitions does not 

exactly equal the total number of unit partitions, add 1 to the 

number of partitions in the first field in the composite key for 

a ' b ' c a ' b b ' a ' c c a ' c c ' b 60 the fact table. All constraints, including the total number of 



partitions, can not be violated. Most important of all, the 

Note that all the fields in the composite key do not product of the partitions in each of the fields must be equal 

necessarily have to be used and the fields can be used in any to or less than the total number of unit partitions required to 

order. store the data from the fact table. 

Step 2) is to determine the number of partitions for each 65 2) Continue with each field until the last field in the 

field that will be required. This step is slightly more com- composite key or bottom level in the hierarchy has been 

plex. increased by one partition or until the product of partitions 
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equals the total unit partitions. Proceed as in the first part of the right in the list or vice versa. If the correct key boundary 

the technique with one exception. The partitions are struc- can not be found in the list, then the boundary key is between 

tured in hierarchical fashion as described in the above the last two keys in the list that were checked. In this case, 

section on structure. In all but the first field or top level in keys values from all the partition area which are between the 

the hierarchy, a partition must be added to each sub-partition 5 fai two key values in the list are checked. This can also be 

group with respect to its parent level to increase the overall d one m an efficient manner with a binary search since the 

partitions for the level by one. partition areas are sorted. Notice as more checking is 

3) If the product of partitions is not equal to the total performed, the range of possible values becomes more 

number of unit partitions at this point, add one to each narrow 

sub-partition group in the bottom most level of the partition 10 A . , . 

hierarchy until the number of partitions in the hierarchy is Assuming that the fact table contains only distinct key 
exactly equal to the total number of unit partitions. Partitions values > mere ^ exactl y one ke y somewhere in the data 
should be added from left to right if implied addressing is to partitions the data correctly. Once all partitions are 
be maintained. determined in this manner, data is moved to the correct 
In most cases, some free space must be left in each 15 partition. Again, all this can be done in parallel since each 
partition because record sizes do not always align perfectly partition boundary can be determined independently, 
with partition sizes. As a result, free space up to size of the Both the sorting and parallel partitioning algorithms can 
maximum length of a record must be left vacant. This must be used to determine if there are duplicate keys in the fact 
be considered when calculating the number of unit partitions ta t> le sorting, of course, duplicate key values will be 
required to store a fact table. If the records are fixed length, 20 relocated to adjacent positions. With the parallel partitioning 
then the sum of available space on a unit partition minus any technique, duplicate keys are placed in the same unit parti- 
partition overhead such as a field containing the number of ^ whefe m ^ ^ easfl identified b ^ me data 
records per partition is divided by the record size and each ^ ^ whicfa wqM be done for 

truncated to a whole number. This determines the number or - 

. . , .... . ■* pertormance reasons, 

records that can be packed into each unit partition. This 25 r 

number can be divided into the total number of records in the Partitions often are not the same size. Some of the 

fact table and rounded up to the next highest integer to partitions may contain exacdy one additional unit partition, 

determine the number of unit partitions that are required. For The partitions with extra unit partitions should all be moved 

variable length records, the unit partition size minus the to one side or the other (usually the left) so that Hypergrids 

maximum length of a variable length record in addition to 30 not using pointers can easily calculate or imply addresses 

any other overhead must remain vacant on each partition. within the data structure. 

The total size of the fact table in bytes is divided by this steps 5) and 6) are to repeat the process in Steps 3) and 

adjusted unit partition size to determine how many unit 4 ) for each su b- pa rtition on the next lower level. Of course, 

partitions will be required to store the data. This vacant area now ^ or parallel partitioning is limited to the scope 

is used as an overflow on each unit partition in the case of 35 of mdividual sub . part i t ions. The process is the same except 

variable length records. mat me second field m the comr^site is used as Uie leading 

Steps 3) andj^are to partition the data by the first field fidd for me key§ M ^ m me key ^ 

10 TOs°^te done with at least a couple of different f 1 need f° ^ je cond field is brought to the front 
methods. One method is to sort the data by the composite 40 b * amoving the first field and concatenating it on the back 
key with the first field as the lead field in the key. For large of ^P 0 ^ key- 
data, modem computer technology allows parallel sorting Step 7) is to repeat Steps 5) and 6) until the data is 
techniques to be used to reduce the time required to sort the partitioned by all the composite key member fields, 
data. Once the data is sorted in this fashion, it can easily be Step 8) is to build a non-dense index for all composite key 
partitioned according to the number of partitions in the first 45 fields. One non-dense index is built for each member field or 
field. Alternatively, the data can be partitioned without level in the partition hierarchy. Each unit partition boundary 
completely be sorted. Instead, each partition area is sorted is in exactly one B-tree. No unit partition boundary is in 
separately and an efficient partitioning algorithm is used to more than one B-tree. The top level non-dense index con- 
find the boundary key values for each partition that would be tains all of the most coarse grained partitions. The next level 
in place if the data were completely sorted. This can also be 50 contains the sub-partitions of the first level partitions. Each 
performed in parallel. The data in each partition area is of the partition level boundaries is contained in one non- 
sorted according to the composite key with the leading key dense index, 
as the first field in the key. In searching for the first, second, 

third, or nth partition boundary value, the boundary value is ^ 

obtained independently for each partition. These indepen- 55 g£E 1 ) 

dent partitions values are sorted and assembled into a list of ^ ^ ^ ^ ^ ^ dcsigMtc ^ 

potential boundary Values. (There should be n squared key product, and time as the key positions for the composite key in the 

boundaries in the list where n is the number of partitions for previous example), 

the first member field or highest level partition.) Once this 2 ) 

list is known the boundary value for the overall list is 60 ^ aumbcr of partitiona ^ ^ (cg> 1Q for sUUc 2for 

Contained in the closed and discrete interval which IS COV- product, and 4 for time). This might involve an analysis data step to 

ered by the these keys. It is simply a matter of narrowing determine the number of partitions per key. 

down the list. To narrow it down, each key is checked to Sle P 3 ) 

determine how it partitions all the data. This can be done by ~. , . . . _» . ... . t . . . 

....... , _ , . J Order all data to be stored using this concatenated key by ascending 

Summing Up the mdividual number Of records on each Side 65 or descending order with the key positions specified in Step 1). 

of the key for all the other parition areas. If the key partitions This step is not needed for parallel partitioning, 
the data too far to the left according to sort order, move to 
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Step 4) 

Divide the data into the number of partitions specified for the first 
key according to Steps 1) and 2). 
Step 5) 

Sort the data within each first key partition according to the remaining 
keys (e.g., if the data is partitioned by state, sort the data by product 
and time within each state). 
Step 6) 

Divide the data into the number of partitions specified for the second key 
as was done in Step 4). 
Step 7) 

Repeat Steps 5) and 6) for all remaining key fields in the composite key. 
Step 8) 

Build the B-trees associated with each set of keys associated with the 
composite key. There will be one B-tree index per component key. 
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Querying the Hypergrid 20 

The steps detailed below correspond to the data flow 
diagram that follows them. 

Step 1) is to produce a set of selection keys from user 
query criterion. This can be in the form of keys that are to 
be included from each dimension or key field, keys that are 25 
excluded from each dimension or key field, or no selection 
criteria for a dimension or key field which translates into all 
the keys for the dimension. 

Step 2) is to order the selection keys for each dimension 



present the query results. Full sorting should not be neces- 
sary to aggregate and reduce data. 



Step 1) 

Produce a set of selection keys from the user query. For each component 
key, this should include either a set of selected keys, a set of 
excluded keys, or a set of all keys. 
Step 2) 

Order the selection keys for each dimension by the same ascending/ 
descending order as the corresponding key field in the Hypergrid. 
Step 3) 

Divide the query up according to the partitions of the first key 
in the composite key of the data. 
Step 4) 

Divide the query according to the sub-partitions of the next key 
field in the composite key. 
Step 5) 

Repeat Steps 3) and 4) until the query is completely partitioned. 
Step 6) 

As the sub queries from each partition are being concatenated together, 
merge and aggregate the data (if necessary). 



Description — Alternative Embodiments 
Hypergrid for Time Series Data 

Virtually all OLAP data is time series data. That is, 
different data for the same composite key fields other than 
according to the order of the component key fields in the 30 me key field occur in different periods in time. For 
Hypergrid. example, in March of 1995 fields A=2 and B=4 may not be 

Step 3) is to divide the query up according to the partitions represented m a f act febfe because the corresponding metric 
defined in the first level of the composite key of the fidds m Q 0f Qul] fc JuQe of 1996 M of 1996 and 
Hyper^id^js is done via the non-dense index correspond- December of 1996 A=2 ^ B „ 4 ^ represented ^th a 
ing to the first level in the composite key. If only some of the „„ , f r • a* 1 

keys are selected from the corresponding dimension, then 35 ? on ^!° Rvalue. In fact in time series data, particular 
only the partitions associated with the selected keys are kG J va } ues (^eluding the time key) are likely to occur very 
searched. If an exclusion list is used, only the partitions with often throughout all the time periods in the data, 
exclusion keys are filtered for records that will not be In addition, once data is cleansed, validated, and placed m 

included in the results of the query. In this case, all other a fact uble corresponding to a specific time period, it is often 
partitions are included in their entirety. If neither selection 40 frozen and not changed or updated anymore. After the life 
nor exclusion are employed, all partitions are searched. The cycle of the data is complete it is archived or removed from 
star-query method technique can efficiently be used here. the fact table. 

With this technique, all possible composite key values A simple modification to the Hypergrid allows it to 

specified by the query criteria are divided into the partitions support this scenario very well. The data is partitioned 
to make the query search more efficient. Note also, that the 45 according to time periods. These time periods can be fixed, 



complete list of composite search keys generated by the 
query criteria need not be generated to execute the search. 
The list can be treated as a virtual list and divided into the 
appropriate partitions and the query can correspondingly 
move through the query hierarchy. 

Step 4) is to divide the query up according to the sub- 
partitions in the next level of the composite key in the 
Hypergrid. This is done according to the non-dense index 



such as months, or the amount of data that is required to fill 
an I/O device or group of I/O devices to capacity. Once a 
partition or device is full, data with new or later time keys 
are moved to the next partition or device. Anon-dense B-tree 
50 is maintained to stored boundary key values for the time 
partitions. As in the standard Hypergrid partitions, the time 
partition boundaries may need to make use of other fields to 
provide exact partition boundaries. 

Within each partition, data is stored according to the 



associated with the next level key field in the Hypergrid. The 

partitions are searched according to the techniques in Step 55 standard Hypergrid data structure detailed above. 

3). Hypergrid for \folalile Data 

Step 5) is to repeat Step 3) and Step 4) until the search The standard Hypergrid may be inappropriate for large 
space is completely partitioned. At this point, only the volatile fact tables. If fixed size fields are used, metric values 
necessary unit partition needs to be retrieved from secondary can be updated without reorganizing the data. But, if van- 
storage. Each of these unit partitions are searched according 60 able length fields are used or if records are inserted or 



the query selection criteria and the boundaries of the unit 
partition. If a nested Hypergrid structure is used, repeat the 
Hypergrid search method for the data in each unit partition. 
All partitions can be searched in parallel with respect to both 
processors and secondary storage resources. 

Step 6) is to merge and aggregate query results if 
necessary, and perform any remaining processing needed to 



65 



deleted, a Hypergrid associated with a fact table must be 
reorganized. This is a difficult process for very large fact 
tables (perhaps larger than 1 terabyte) that are updated 
frequently. 

The Hypergrid can be modified to better accommodate 
this situation. Each I/O device can be treated like a data 
bucket or partition in a non-dense B-tree, B+-tree, or B# tree 
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(like a B-tree but maintains a minimum of % full data matrix formed by all possible combinations of the composite 

buckets). Newly inserted or deleted data causes the data on key values) the data can be stored in a multidimensional data 

each I/O device to be reorganized. This is a much easier task structure such as depicted in FIG. 1. In this situation, all 

than reorganizing the entire Hypergrid for cases when the elements of the multidimensional matrix are stored in a flat 

fact table is very large. If a partition contains too much or too 5 file in a regular manner so that each element can be directly 

little data after inserts or deletes, data is redistributed among accessed with only one I/O or seek operation when queries 

the partition and its neighbors. Partitions) are split or, ™ performed. Id FIG. 1 a ^file ^containing 90 elements could 

partitions) are merged. The disadvantage is that partitions be crea * d ^presenting the 6 months, 5 products, and 3 

are no longer completely full. According to test results on, s ™?**°™ m lhe Uofortunatel y> howevcr > most 

^ j^« L t ' OLAP data is sparse by nature. 

B+-trees, and B# trees, the partitions average between 75 10 „ r / _ . , A . . . 

and 80% capacity Consider an example of such data. Assume that the data 

a a. %- o • j-c *• -,u- u is sparse so that there are missing values. Further assume 

As wuh the Time Scnes modification, withm each tt J , UBdaUi5salesdatalIldlhit g 0OllUbe by state , 

partihon data .sstored and organized according the standard and monthasshown below in pic. 2A and FIG. 2B. 

Hypergnd method detailed above. P, G 2C represents a millimum M of sorted copies of the 

Operation-Alternative Embodiment 15 data .'!' at ™?J? ta K ^ d t0 efficiendy query on all 

u 'af t « ■ n possible muludimension key combinations. In spite of the 

Hypergna tor lime series iJata . ... , fact that this data is sparse and irregular, it needs only to be 

To build the Time Series Hypergnd, determine the size of stQred fa a Hypergrid once. If such a table contains 100,000 

the time partition and whether it is associated with an I/O by me three dimensions above, and 

device or a fixed time period such as a month or year. Once 20 each ^ Wock holds 100 records> the ^ 

the time partition is determined, add data to the most current partitioned such that the data is divided into 10 different 

time partition as it is validated and cleansed. Optionally, rangcs according to what state it represents, 10 different 

when the oldest time partition becomes outdated, archive or product ranges within the state partition, and 10 different 

remove it from the Hypergrid. Within each partition, orga- time ranges within the combination of state/product ranges, 

nize and store the data with the standard Hypergrid method. 25 The order of the partitioning is predetermined by the admin- 

Since the data is already partitioned by time, it is not istrator of the data warehouse. The order in this case could 

necessary to use time as a member field in the individual be state, product, and time. The table is first sorted and 

time partitions. The boundary key values corresponding to partitioned according to state. Therefore the entire table is 

each time partition are added to the Hypergrid B-tree divided into 10 distinct state partitions. Then it is sorted and 

indexes as the highest level B-tree index. As is the case with 30 partitioned into 10 different product partitions within each 

the standard Hypergrid method, if equal sized I/O devices state. Likewise, 10 distinct partitions of time can be created 

are used as the time partitions, the other member fields of the within each product partition. If a key does not change value 

composite field will be needed to determine precise key exactly at the partition point, lower order columns in the 

boundaries. This is not true if exact time partitions such as composite key can be used to create a precise partition 

months or years are used. The time field is designated as the 35 division point. After all these partitions are created, the 

leading or most significant field. 100,000 records are neatly divided into exactly 1000 pages 

To query the Time Series Hypergrid proceed exactly as with approximately 100 records on each block. Non-dense, 

with the standard Hypergrid. B-tree indexes can be used to provide indexes into the 

Hypergrid for Volatile Data partitions. One index involving all key columns in key order 

To build the Volatile Data Hypergrid, treat each I/O device 40 and with partition pointers can be used as the primary index, 

as a data bucket would be treated in a B-tree, B+-tree, or B# Then one index for each key column with fewer partitions 

tree. Thus data can be sorted according to the composite key can be used for each of the other key columns. Each higher 

and massed loaded, or it can be batch loaded in an incre- level index would have fewer keys corresponding to fewer 

mental fashion. To load the data incrementally, add data one partitions. The internal structure would be similar to FIG. 3. 

record at a time until one of the I/O device partitions become 45 The partitions created by the Hypergrid are designed to 

full. When this happens, sort the data in the I/O device improve the efficiency of I/O operations by concentrating 

according to the boundary key order and split the data in the closely related data items together. Data can be partitioned 

I/O device into two I/O devices, or three if B#- trees are used. by the manner explained above across disk devices or other 

Similarly when the number of records fall below minimum hardware devices as depicted in FIG. 4A, FIG. 4B, FIG. 4C, 

thresholds for the I/O device, merge the records in the device 50 and FIG. 4E. Data on each hardware device can be further 

with an adjacent device according to key order. When all partitioned across all disks, disk pages, and/or disk cylinders 

new data in the Volatile Data Hypergrid is updated, structure within each hardware partition as depicted in FIG. 4A, FIG. 

the data within each I/O device partition according to the 4c, FIG. 4D, and FIG. 4E. FIG. 4E represents a RAID I/O 

standard Hypergrid method. As with the other forms of storage device where several disks simulate one virtual disk. 

Hypergrids, all fields in the composite key of the fact table 55 i n this case, the device is treated as one virtual disk. In 

must be used to determine exact key boundaries. Finally, addition, the partitioning along with proper partitioning of 

within each I/O device partition, the leading key of the I/O nested star-joins clusters needed I/Os together very effi- 

partition need not be used in partitioning the data since it has cicntly by grouping I/O requests together and by accessing 

already been partitioned by this key. only the partitions that are needed as indicated by the 

Queries can be performed as they are in the standard 60 partitions with hatching in FIG. 4C and FIG. 4D. Finally, the 

Hypergrid method above. non-dense indexing that is used by the partitioning works 

very well with I/O caching such as is found in many high 

Drawing bigures performance I/O subsystems. Non-dense indexes are also 

OLAP (Online Analytical Processing) or multidimen- usually small enough to be completely contained in RAM 

sional type data is stored in several dimensions or via a 65 memory, cl Summary, Ramifications, and Scope 

composite key. In an ideal situation where there is no This invention truly has the ability to revolutionize the 

sparsity in the data (no missing data in the multidimensional data warehouse and OLAP industry. It is simple to imple- 
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ment and maintain yet provides very powerful features 
designed just for this type of data. 

It uses much less computer storage space. Since OLAP 
and data warehouse applications are well known to use 
copious amounts of computer storage space, this is a great 
advantage. It can pack file pages or unit partitions near 100% 
full and does not require any additional index space. Due to 
the ability of this invention to pack file pages or unit 
partitions to nearly 100% capacity, the structure can be 
nested so that one hypergrid contains unit partitions which 
contain another hypergrid. This further aides query speed 
once data is moved into computer memory. 

It can take advantage of very fine grained parallelism at 
the storage device level and can take advantage of full 
multidimensional partition selectivity simultaneously. This 
"divides and conquers" data search space to a maximum 
degree. 

It can create a perfect n-dimension hypercube no matter 
how sparse the data or how many multidimensional keys 
there are in the data. 

It provides maximum data locality. All records that are 
being queried are extracted from a file page or unit partition 
at one time. No file page or unit partition is ever retrieved 
more than once during the life of a query. If all keys in the 25 
multidimension composite key are specified, exactly one I/O 
operation is required to retrieve the record. 

Nested star loops associated with OLAP type queries can 
be perfectly unfolded to match the exact structure of data 
stored by the method presented in this invention. 

It is very flexible. Weights or other constraints can be 
added to adjust the degree of symmetric multidimensional 
access of individual keys within a multidimension compos- 
ite key. 

As explained in the alternative embodiments section 
above, it provides good bulk load and update performance 
for even very volatile data. As further explained, it provides 
especially good bulk load and update performance for true 
time-series data. 

The description above contains many specificities. But, 
these should not be used to limit the scope of this invention. 
Instead, they provide illustrations of some of the possible 
embodiments of the invention. The invention can be used in 
a variety of other ways. It can be used with variable length 
keys, other build and query techniques, text keys, in com- 
bination with other data structures, etc. 

As a result the scope of this invention should be deter- 
mined by the appended claims and their legal equivalents, in 
lieu of the descriptions and specifications above. 

I claim: 

1. A method of partitioning data regardless of the sparsity 
of said data on to computer memory or computer storage 
media with one or more keys or groups of keys from said 
data such that the selectivity produced by any combination 
of said keys or said groups of one or more keys can be 
precisely controlled comprising the steps of: 

a) determining the number of records from said data to be 
packed into each unit partition 

b) determining selectivity of each said key or said group 
of one or more keys chosen from said composite key so 
that the combined selectivity of all said keys or said 
groups of one or more keys is equal to the total number 
of unit partitions in said data 
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d) partitioning said data by said first key or group of one 
or more keys of said chosen component keys possibly 
followed by one or more of the other component keys 
in said composite key 

e) creating a non-dense index formed from said first key 
or group of one or more keys and possibly followed by 
one or more of the other component keys in said 
composite key aligned with the boundaries of said 
partitions created in step d) 

f) choosing a new first key or group of one or more keys 
from said chosen component keys in said composite 
key 

g) repeating steps d), e), and f) within each said partition 
created from the previous iteration of step d) as many 
times as required to produce said total number of unit 
partitions contained within said data thereby producing 
an organization of said data whose said selectivity of 
combinations of said keys can be precisely controlled 
by said selectivity for each said key or group of one or 
more said keys as determined in step b). 

2. A method of partitioning data regardless of the sparsity 
of said data on to computer memory or computer storage 
media with one or more keys or groups of keys from said 
data such that the selectivity of any combination of involved 
said keys or said groups of one or more keys is symmetric 
yet the degree of packing of records from said data into unit 
partitions can be precisely controlled comprising the steps 
of: 

a) determining the number of records from said data to be 
packed into each unit partition 

b) determining selectivity of each said key or said group 
of one or more keys chosen from said composite key so 
that the combined selectivity of all said keys or said 
groups of one or more keys is equal to the total number 
of unit partitions in said data 

c) arbitrarily choosing a first component key or group of 
one or more component keys from said chosen com- 
ponent keys in said composite key 

d) partitioning said data by said first key or group of one 
or more keys of said chosen component keys possibly 
followed by one or more of the other component keys 
in said composite key 

e) creating a non-dense index formed from said first key 
or group of one or more keys and possibly followed by 
one or more of the other component keys in said 
composite key aligned with the boundaries of said 
partitions created in step d) 

f) choosing a new first key or group of one or more keys 
from said chosen component keys in said composite 
key 

g) repeating steps d), e), and f) within each said partition 
created from the previous iteration of step d) as many 
times as required to produce said total number of unit 
partitions contained within said data thereby producing 
an organization of said data whose said selectivity for 
any combination of said keys or said groups of one or 
more keys can be made to be symmetric by step b) and 
whose packing of said records into said unit partitions 
can be precisely controlled by step a). 

3. The method of claim 2, wherein all said unit partitions, 
file pages, or I/O sectors are fully packed to hold the 
maximum number of said records per unit partition. 

4. The method of claim 3, wherein said records within 
each said unit partition are further subpartitioned by the 



c) arbitrarily choosing a first component key or group of 65 method of claim 3 in a nested fashion such that additional 
one or more component keys from said chosen com- said computer memory or said computer storage media are 
ponent keys in said composite key not required. 
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5. The method of claim 4 combined with other data 
structures or partitioning methods such that said data can be 
partitioned across hardware devices or entire computers with 
conventional methods and subpartitioned by the method of 
claim 3 in a nested fashion such that additional said com- 5 
puter memory or said computer storage are not required. 

6. A method of partitioning data regardless of the sparsity 
of said data on to computer memory or computer storage 
media with one or more keys or groups of keys from said 
data such that the selectivity of any combination of involved 10 
said keys or said groups of one or more keys is symmetric 
yet a non-dense index which requires no addition computer 
memory or computer storage media space can be built 
comprising the steps of: 

a) determining the number of records from said data to be 15 
packed into each unit partition taking into account the 
extra available space in said unit partitions due to the 
removal of one or more said component records from 
one said record in said unit partitions 

b) detennining selectivity of each said key or said group 20 
of one or more records chosen from said composite key 

so that the combined selectivity of all said keys or said 
groups of one or more keys is equal to the total number 
of unit partitions in said data 

c) arbitrarily choosing a first component key or group of 
one or more component keys from said chosen com- 
ponent keys in said composite key 
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d) partitioning said data by said first key or group of one 
or more keys of said chosen component keys possibly 
followed by one or more of the other component keys 
in said composite key 

e) creating a non-dense index formed from said first key 
or group of one or more keys and possibly followed by 
one or more of the other component keys in said 
composite key aligned with the boundaries of said 
partitions created in step d) and removing one or more 
said component keys from one said record out of each 
said unit partition corresponding to each entry in said 
non-dense index 

f) choosing a new first key or group of one or more keys 
from said chosen component keys in said composite 
key 

g) repeating steps d), e), and f) within each said partition 
created from the previous iteration of step d) as many 
times as required to produce said total number of unit 
partitions contained within said data thereby producing 
an organization of said data whose said selectivity for 
any combination of said keys or said groups of one or 
more keys can be made to be symmetric by st b) and on 
which a corresponding non-dense index requiring no 
additional computer memory or computer storage 
media space can be built. 



